Transition from E+ 23.2.0 to E+ 24.1.0
Foreword
-
Part 1:
aims to transition all regression tests from one E+ version to the next, and allows you to run each test in both the old and the new version
- These will take quite some time to run (about 1hr to run the tests in the OLD OpenStudio version, transition the IDFs to the new E+ version and run them, and run the tests in the NEW OpenStudio Version, based on almost 200 files currently)
- By default it will just copy over the SQL from the regression test to place in OLD_DIR, but if you want to force rerun the IDF in the old E+ version you can.
- At the end of Part 1, you will have three CSV files, one per version, with the site KBTUs for each test. And you also have an organized tree of VERSION/TEST_NAME/ output directories that have the SQL files we will use for sections 6+.
-
Part 2:
aims to analyze the differences between versions
- Section 3.1 just re-queries all SQL file (or you can reload the three CSV files) to highlight the tests with the biggest site KBTU differences
- Section 3.2 provides a high-level interface that only requires to pass a test name and it will query the relevant SQL files and produce visualization (tables, grouped bar charts, and heatmaps) to analyze where differences may be coming from
PART 1: Transition all regression tests and run them in both E+ versions ¶
Part 1 is ommited in the HTML because the interesting part is the output and analysis at the end
PART 2: Analyzing differences ¶
Analyzing E+ runtime ¶
THIS NO LONGER WORKS, last version where it worked was 9.2.0... I don't know where the runtime went in the SQLFile
Concat frames and look at files that failed to run ¶
These are the files were we have some (but not all) failures
| E+ | 23.2.0-7636e6b3e9 | 24.1.0-9d7789a3ac | |
|---|---|---|---|
| OS | 3.7.0 | Transition | 3.8.0 |
| python_plugin.osm | 1044617.682811 | nan | nan |
| python_plugin.rb | 1044617.682811 | 1044617.682811 | nan |
| storage_liion_battery.osm | 2089235.365622 | nan | 2089235.365622 |
| storage_liion_battery.rb | 2089235.365622 | nan | 2089235.365622 |
Rerun transition files that need expand objects (can't do it in parallel) ¶
First look at where we have deviations ¶
Setting % diff threshold to 0.010%
Deviations in Transition and/or new OpenStudio ¶
Deviations from Transition to new OS only ¶
Deviations from Transition to new OS only: Differences above 0.010%
| E+ | 24.1.0-9d7789a3ac | |
|---|---|---|
| OS | Transition | 3.8.0 |
| vrf_fluid_temperature_control.rb | 0.004% | -0.011% |
| vrf_fluid_temperature_control.osm | 0.004% | -0.011% |
| multi_stage_electric.rb | -0.073% | 0.165% |
| multi_stage.rb | -0.166% | -0.014% |
| multi_stage_electric.osm | -0.270% | 0.225% |
| centralheatpumpsystem.osm | -0.557% | -0.011% |
| centralheatpumpsystem.rb | -0.777% | -0.010% |
Grouped bar chart of differences compared to the old OpenStudio ¶
How to read this chart :
The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results.
What you need to pay special attention to is when you don't have the same difference between the Transition to Old OS and the New OS to old OS (meaning the difference is not E+'s fault, but OpenStudio's).
| OS | Transition | 3.8.0 |
|---|---|---|
| centralheatpumpsystem.rb | -0.78% | -0.79% |
| centralheatpumpsystem.osm | -0.56% | -0.57% |
| heatpump_airtowater_fuelfired.osm | 0.37% | 0.37% |
| heatpump_airtowater_fuelfired.rb | 0.37% | 0.37% |
| lowtemprad_constflow.osm | 0.35% | 0.35% |
| daylighting_no_shades.osm | 0.30% | 0.30% |
| daylighting_shades.osm | 0.29% | 0.29% |
| photovoltaics.osm | 0.29% | 0.29% |
| pv_and_storage_facilityexcess.osm | 0.29% | 0.29% |
| transformer.osm | 0.29% | 0.29% |
| unitary_systems_airloop_and_zonehvac.osm | 0.29% | 0.29% |
| plantloop_avms.osm | 0.29% | 0.29% |
| epw_design_conditions.osm | 0.29% | 0.29% |
| additional_props.osm | 0.29% | 0.29% |
| pv_and_storage_demandleveling.osm | 0.29% | 0.29% |
| moisture_settings.osm | 0.29% | 0.29% |
| utility_bill02.osm | 0.29% | 0.29% |
| utility_bill01.osm | 0.28% | 0.28% |
| interior_partitions.osm | 0.27% | 0.27% |
| solar_collector_flat_plate_water.osm | 0.27% | 0.27% |
| multi_stage_electric.osm | -0.27% | -0.05% |
| adiabatic_construction_set.rb | 0.27% | 0.27% |
| solar_collector_flat_plate_photovoltaicthermal.osm | 0.26% | 0.26% |
| solar_collector_integralcollectorstorage.osm | 0.26% | 0.26% |
| water_heaters.osm | 0.26% | 0.26% |
| swimmingpool_indoor.osm | -0.24% | -0.24% |
| swimmingpool_indoor.rb | -0.24% | -0.24% |
| surface_properties_ground_and_solarmult.rb | 0.19% | 0.19% |
| surface_properties_ground_and_solarmult.osm | 0.19% | 0.19% |
| surface_properties_lwr.rb | 0.18% | 0.18% |
| surface_properties_lwr.osm | 0.18% | 0.18% |
| multi_stage.rb | -0.17% | -0.18% |
| adiabatic_construction_set.osm | 0.18% | 0.18% |
| outputcontrol_files.osm | 0.18% | 0.18% |
| outputcontrol_timestamp.rb | 0.18% | 0.18% |
| outputcontrol_files.rb | 0.18% | 0.18% |
| output_objects_2.osm | 0.18% | 0.18% |
| outputcontrol_timestamp.osm | 0.18% | 0.18% |
| output_objects_2.rb | 0.18% | 0.18% |
| photovoltaics_sandia.rb | 0.18% | 0.18% |
| photovoltaics_sandia.osm | 0.18% | 0.18% |
| output_objects.osm | 0.17% | 0.17% |
| output_objects.rb | 0.17% | 0.17% |
| window_property_frame_and_divider.osm | 0.17% | 0.17% |
| window_property_frame_and_divider.rb | 0.17% | 0.17% |
| foundation_kiva_customblocks.osm | 0.17% | 0.17% |
| foundation_kiva.osm | 0.17% | 0.17% |
| unitary_system.rb | 0.16% | 0.16% |
| unitary_system.osm | 0.16% | 0.16% |
| pthp_othercoils.osm | 0.15% | 0.15% |
| zone_hvac.rb | 0.15% | 0.15% |
| zone_hvac_equipment_list.osm | 0.15% | 0.15% |
| zone_hvac_equipment_list.rb | 0.15% | 0.15% |
| zone_property_user_view_factors_by_surface_name.osm | 0.14% | 0.14% |
| unitary_vav_bypass_coiltypes.rb | 0.13% | 0.14% |
| unitary_test.osm | 0.13% | 0.13% |
| unitary_test.rb | 0.13% | 0.13% |
| unitary_vav_bypass_coiltypes.osm | 0.13% | 0.13% |
| coil_waterheating_desuperheater.osm | 0.13% | 0.13% |
| multi_stage.osm | -0.12% | -0.11% |
| coilsystem_dxhx.rb | 0.12% | 0.12% |
| coilsystem_dxhx_desiccant_balancedflow.osm | 0.12% | 0.12% |
| coilsystem_dxhx_desiccant_balancedflow.rb | 0.12% | 0.12% |
| coilsystem_dxhx.osm | 0.12% | 0.12% |
| generator_microturbine.osm | 0.12% | 0.12% |
| heatexchanger_desiccant_balancedflow.rb | 0.12% | 0.12% |
| heatexchanger_desiccant_balancedflow.osm | 0.12% | 0.12% |
| roof_vegetation.osm | 0.11% | 0.11% |
| roof_vegetation.rb | 0.11% | 0.11% |
| exterior_equipment.osm | 0.11% | 0.11% |
| exterior_equipment.rb | 0.11% | 0.11% |
| space_load_instances.osm | 0.10% | 0.10% |
| space_load_instances.rb | 0.10% | 0.10% |
| heatpump_varspeed.osm | 0.10% | 0.10% |
| heatpump_varspeed.rb | 0.10% | 0.10% |
| dsn_oa_w_ideal_loads.rb | 0.10% | 0.10% |
| shadingcontrol_singlezone.osm | 0.10% | 0.10% |
| shadingcontrol_singlezone.rb | 0.10% | 0.10% |
| daylighting_devices.rb | 0.10% | 0.10% |
| ExampleModel.rb | -0.09% | -0.09% |
| lifecyclecostparameters.rb | -0.09% | -0.09% |
| multi_stage_electric.rb | -0.07% | 0.09% |
| unitary_vav_bypass_plenum.osm | 0.09% | 0.09% |
| baseline_sys02.osm | 0.08% | 0.08% |
| chillers_tertiary.osm | 0.08% | 0.08% |
| chillers_tertiary.rb | 0.08% | 0.08% |
| dsn_oa_w_ideal_loads.osm | 0.08% | 0.08% |
| humidity_control.rb | 0.08% | 0.08% |
| water_economizer.rb | 0.07% | 0.07% |
| unitary_vav_bypass.osm | 0.07% | 0.07% |
| unitary_vav_bypass.rb | 0.07% | 0.07% |
| zone_control_contaminant_controller.rb | 0.07% | 0.07% |
| heatpump_plantloop_eir.osm | 0.07% | 0.07% |
| heatpump_plantloop_eir.rb | 0.07% | 0.07% |
| zone_control_contaminant_controller.osm | 0.07% | 0.07% |
| humidity_control.osm | 0.07% | 0.07% |
| surfacecontrol_moveableinsulation.osm | 0.07% | 0.07% |
| surfacecontrol_moveableinsulation.rb | 0.07% | 0.07% |
| coil_cooling_dx.rb | 0.07% | 0.07% |
| coil_cooling_dx.osm | 0.07% | 0.07% |
| airloop_and_zonehvac.osm | 0.07% | 0.07% |
| generator_windturbine.rb | 0.07% | 0.07% |
| generator_windturbine.osm | 0.07% | 0.07% |
| sizing_zone_dszad.osm | 0.07% | 0.07% |
| sizing_zone_dszad.rb | 0.07% | 0.07% |
| shadowcalculation.osm | 0.07% | 0.07% |
| shadowcalculation.rb | 0.07% | 0.07% |
| coil_waterheating_desuperheater.rb | 0.07% | 0.07% |
| daylighting_devices.osm | 0.07% | 0.07% |
| coil_userdefined.rb | -0.07% | -0.07% |
| coil_userdefined.osm | -0.07% | -0.07% |
| zone_hvac2.rb | 0.06% | 0.07% |
| chiller_electric_ashrae205.rb | 0.07% | 0.07% |
| chiller_electric_ashrae205.osm | 0.07% | 0.07% |
| baseline_sys01.osm | 0.07% | 0.07% |
| pvwatts.osm | 0.07% | 0.07% |
| baseline_sys03.rb | 0.07% | 0.07% |
| hot_water.rb | 0.07% | 0.07% |
| evaporative_cooling.rb | 0.07% | 0.07% |
| ideal_plant.rb | 0.07% | 0.07% |
| ideal_plant.osm | 0.07% | 0.07% |
| hot_water.osm | 0.07% | 0.07% |
| air_chillers.osm | 0.06% | 0.06% |
| zone_hvac2.osm | 0.06% | 0.06% |
| availability_managers.osm | -0.06% | -0.06% |
| baseline_sys06.rb | 0.06% | 0.06% |
| fuelcell.rb | 0.06% | 0.06% |
| fuelcell.osm | 0.06% | 0.06% |
| surface_properties.osm | 0.06% | 0.06% |
| pv_and_storage_demandleveling.rb | 0.06% | 0.06% |
| baseline_sys08.rb | 0.06% | 0.06% |
| utility_bill01.rb | 0.06% | 0.06% |
| vrf_watercooled.osm | 0.06% | 0.06% |
| daylighting_no_shades.rb | 0.06% | 0.06% |
| ems.rb | 0.06% | 0.06% |
| epw_design_conditions.rb | 0.06% | 0.06% |
| additional_props.rb | 0.06% | 0.06% |
| plantloop_avms.rb | 0.06% | 0.06% |
| baseline_sys01.rb | 0.06% | 0.06% |
| pvwatts.rb | 0.06% | 0.06% |
| transformer.rb | 0.06% | 0.06% |
| photovoltaics.rb | 0.06% | 0.06% |
| utility_bill02.rb | 0.06% | 0.06% |
| daylighting_shades.rb | 0.06% | 0.06% |
| pv_and_storage_facilityexcess.rb | 0.06% | 0.06% |
| solar_collector_flat_plate_photovoltaicthermal.rb | 0.06% | 0.06% |
| moisture_settings.rb | 0.06% | 0.06% |
| plenums.osm | 0.06% | 0.06% |
| plenums.rb | 0.06% | 0.06% |
| solar_collector_flat_plate_water.rb | 0.05% | 0.05% |
| water_heaters.rb | 0.05% | 0.05% |
| airterminal_cooledbeam.osm | 0.05% | 0.05% |
| solar_collector_integralcollectorstorage.rb | 0.05% | 0.05% |
| availability_managers.rb | -0.05% | -0.05% |
| vrf_airloophvac.osm | 0.04% | 0.04% |
| vrf_watercooled.rb | 0.04% | 0.04% |
| foundation_kiva_customblocks.rb | 0.04% | 0.04% |
| vrf_airloophvac.rb | 0.04% | 0.04% |
| baseline_sys09.rb | 0.04% | 0.04% |
| dist_ht_cl.rb | 0.04% | 0.04% |
| setpoint_manager_systemnodereset.osm | 0.04% | 0.04% |
| setpoint_manager_systemnodereset.rb | 0.04% | 0.04% |
| foundation_kiva.rb | 0.04% | 0.04% |
| headered_pumps.rb | 0.04% | 0.04% |
| baseline_sys04.rb | 0.04% | 0.04% |
| baseline_sys02.rb | 0.04% | 0.04% |
| heatexchanger_airtoair_sensibleandlatent.rb | 0.04% | 0.04% |
| solar_collector_flat_plate_photovoltaicthermal_bipvt.osm | 0.04% | 0.04% |
| solar_collector_flat_plate_photovoltaicthermal_bipvt.rb | 0.04% | 0.04% |
| refrigeration_system_2.rb | 0.04% | 0.04% |
| refrigeration_system_2.osm | 0.04% | 0.04% |
| baseline_sys10.rb | 0.04% | 0.04% |
| airloop_avms.osm | 0.04% | 0.04% |
| airloop_avms.rb | 0.04% | 0.04% |
| ptac_othercoils.osm | 0.04% | 0.04% |
| zone_fan_exhaust.rb | 0.04% | 0.04% |
| fan_on_off.rb | 0.04% | 0.04% |
| refrigeration_system.osm | 0.04% | 0.04% |
| refrigeration_system.rb | 0.04% | 0.04% |
| fan_componentmodel.rb | 0.04% | 0.04% |
| fan_componentmodel.osm | 0.04% | 0.04% |
| air_terminals.rb | 0.04% | 0.04% |
| schedule_file.rb | 0.04% | 0.04% |
| schedule_file.osm | 0.03% | 0.03% |
| unitary_vav_bypass_plenum.rb | -0.03% | -0.03% |
| surface_properties.rb | 0.03% | 0.03% |
| baseline_sys06.osm | 0.03% | 0.03% |
| pthp_othercoils.rb | 0.03% | 0.03% |
| vrf.rb | 0.03% | 0.03% |
| baseline_sys08.osm | 0.03% | 0.03% |
| zone_property_user_view_factors_by_surface_name.rb | 0.03% | 0.03% |
| baseline_sys05.rb | 0.03% | 0.03% |
| plant_op_deltatemp_schemes.rb | 0.03% | 0.03% |
| plant_op_deltatemp_schemes.osm | 0.03% | 0.03% |
| cooling_coils.osm | 0.03% | 0.03% |
| cooling_coils.rb | 0.03% | 0.03% |
| baseline_sys04.osm | 0.03% | 0.03% |
| baseline_sys07.rb | 0.03% | 0.03% |
| thermal_storage.osm | 0.03% | 0.03% |
| meters.osm | 0.03% | 0.03% |
| meters.rb | 0.03% | 0.03% |
| thermal_storage.rb | 0.03% | 0.03% |
| plant_op_schemes.osm | 0.03% | 0.03% |
| plant_op_schemes.rb | 0.03% | 0.03% |
| lowtemprad_constflow.rb | -0.03% | -0.03% |
| plant_op_temp_schemes.rb | 0.03% | 0.03% |
| plant_op_temp_schemes.osm | 0.03% | 0.03% |
| fan_systemmodel.osm | 0.03% | 0.03% |
| absorption_chillers.rb | 0.03% | 0.03% |
| absorption_chillers.osm | 0.03% | 0.03% |
| fan_systemmodel.rb | 0.03% | 0.03% |
| unitary_systems_airloop_and_zonehvac.rb | 0.03% | 0.03% |
| ideal_loads_w_plenums.rb | 0.03% | 0.03% |
| ideal_loads_w_plenums.osm | 0.03% | 0.03% |
| zone_hvac.osm | -0.03% | -0.03% |
| generator_microturbine.rb | 0.03% | 0.03% |
| ducts_and_pipes.osm | -0.02% | -0.02% |
| baseline_sys03.osm | 0.02% | 0.02% |
| dist_ht_cl.osm | 0.02% | 0.02% |
| setpoint_managers.osm | 0.02% | 0.02% |
| setpoint_managers.rb | 0.02% | 0.02% |
| headered_pumps.osm | 0.02% | 0.02% |
| ducts_and_pipes.rb | -0.02% | -0.02% |
| tablelookup.rb | 0.02% | 0.02% |
| tablelookup.osm | 0.02% | 0.02% |
| vrf.osm | 0.02% | 0.02% |
| airloop_and_zonehvac.rb | 0.02% | 0.02% |
| ptac_othercoils.rb | 0.02% | 0.02% |
| coil_waterheating_desuperheater_2.rb | 0.02% | 0.02% |
| coil_waterheating_desuperheater_2.osm | 0.02% | 0.02% |
| heatpump_hot_water.osm | 0.02% | 0.02% |
| heatpump_hot_water.rb | 0.02% | 0.02% |
| zone_fan_exhaust.osm | 0.02% | 0.02% |
| coolingtowers.rb | 0.02% | 0.02% |
| performanceprecisiontradeoffs.osm | 0.02% | 0.02% |
| performanceprecisiontradeoffs.rb | 0.02% | 0.02% |
| airterminal_cooledbeam.rb | 0.02% | 0.02% |
| interior_partitions.rb | 0.02% | 0.02% |
| vrf_fluid_temperature_control_hr.rb | 0.02% | 0.02% |
| vrf_fluid_temperature_control_hr.osm | 0.02% | 0.02% |
| fan_on_off.osm | 0.02% | 0.02% |
| fluid_coolers.rb | -0.02% | -0.02% |
| fluid_coolers.osm | -0.02% | -0.02% |
| dual_duct.osm | 0.02% | 0.02% |
| dual_duct.rb | 0.02% | 0.02% |
| heatexchanger_airtoair_sensibleandlatent.osm | 0.02% | 0.02% |
| baseline_sys09.osm | 0.02% | 0.02% |
| baseline_sys10.osm | 0.01% | 0.01% |
| coilsystem_waterhx.rb | 0.01% | 0.01% |
| coilsystem_waterhx.osm | 0.01% | 0.01% |
| hightemprad.osm | -0.01% | -0.01% |
| hightemprad.rb | -0.01% | -0.01% |
| heatrecovery_chiller.rb | -0.01% | -0.01% |
| heatrecovery_chiller.osm | -0.01% | -0.01% |
| schedule_ruleset_2012_NonLeapYear.rb | 0.01% | 0.01% |
| schedule_ruleset_2012_LeapYear.rb | 0.01% | 0.01% |
| schedule_ruleset_2012_LeapYear.osm | 0.01% | 0.01% |
| schedule_ruleset_2012_NonLeapYear.osm | 0.01% | 0.01% |
| humidity_control_2.rb | 0.01% | 0.01% |
| humidity_control_2.osm | 0.01% | 0.01% |
| schedule_ruleset_2013.rb | 0.01% | 0.01% |
| schedule_fixed_interval.rb | 0.01% | 0.01% |
| schedule_fixed_interval_schedulefile.rb | 0.01% | 0.01% |
| schedule_ruleset_2013.osm | 0.01% | 0.01% |
Stacked bar chart of differences ¶
How to read this chart :
The percentage differences are calculated compared to the Old OpenStudio results for both the transitioned results and the new OpenStudio results. Then I do
% diff New version = % diff New version - % diff Transition
and plot that as a stacked bar chart. The goal is to more clearly see the differences that are due to OpenStudio by removing the differences due to the new E+.
What you need to pay special attention to is when you see % differences for the new OS.
A cursor along with the total % difference between Old OS and New OS is also plotted . Please see the below example to get a better sense of how the graph is constructed
Let's do the actual plotting now: